XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge

Cross-lingual pre-training has achieved great successes using monolingual and bilingual plain text corpora. However, most pre-trained models neglect multilingual knowledge, which is language agnostic but comprises abundant cross-lingual structure alignment. In this paper, we propose XLM-K, a model incorporating knowledge in pre-training. XLM-K augments existing with two tasks, namely Masked Entity Prediction Task Object Entailment Task. We evaluate on MLQA, NER XNLI. Experimental results clearly demonstrate significant improvements over models. The MLQA exhibit the superiority of related tasks. success XNLI shows better transferability obtained XLM-K. What more, provide detailed probing analysis to confirm desired captured our regimen. code available at https://github.com/microsoft/Unicoder/tree/master/pretraining/xlmk.